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DEFENSE  TRAVEL  SYSTEM 

Estimated  Savings  Are  Questionabie  and 
improvements  Are  Needed  to  Ensure 
Functionaiity  and  increase  Utiiization 


Why  GAO  Did  This  Study 


What  GAO  Found 


In  1995,  the  Department  of  Defense 
(DOD)  began  an  effort  to 
implement  a  standard 
departmentwide  travel  system.  The 
Defense  Travel  System  (DTS)  is 
envisioned  as  DOD’s  standard  end- 
to-end  travel  system.  This 
testimony  is  based  on  GAO’s 
September  2006  related  report. 
Today’s  testimony  highlights  GAO’s 
key  findings  with  regard  to  the 
following  objectives:  (1)  Were  the 
two  key  assumptions  made  in  the 
September  2003  economic  analysis 
reasonable?  (2)  Was  DOD  taking 
action  to  ensure  full  utilization  of 
DTS  and  gathering  the  data  needed 
to  monitor  DTS  utilization?  and 
(3)  Has  DOD  resolved  several 
functional  problems  associated 
with  weak  system  requirements 
and  testing?  To  address  these 
objectives,  GAO  (1)  reviewed  the 
September  2003  DTS  economic 
analysis,  (2)  analyzed  DTS 
utilization  data,  and  (3)  analyzed 
DTS  flight  information. 


What  GAO  Recommends 


GAO  made  four  recommendations 
aimed  at  improving  the 
management  of  DTS,  including 
periodic  reports  on  DTS  utilization 
and  resolution  of  inconsistencies  in 
DTS’s  requirements.  DOD 
generally  agreed  with  the 
recommendations  and  described 
efforts  to  address  them.  DOD  also 
strongly  objected  to  the  finding 
that  the  reported  personnel  savings 
were  imrealistic.  However,  DOD 
provided  no  new  data  that  were 
counter  to  GAO’s  findings. 

www.gao.gov/cgi-bin/getrpt7GAO-07-208T. 

To  view  the  full  product,  including  the  scope 
and  methodology,  click  on  the  link  above. 
For  more  information,  contact  McCoy 
Williams  at  (202)  512-9095  or  Keith  Rhodes 
at  (202)  512-6412. 


GAO’s  analysis  of  the  September  2003  DTS  economic  analysis  fotmd  that  the 
two  key  assumptions  used  to  estimate  annual  net  savings  were  not  based  on 
reliable  information.  Two  cost  components  represent  the  majority  of  the 
over  $56  million  in  estimated  net  savings — persormel  savings  and  reduced 
commercial  travel  office  (CTO)  fees.  In  regard  to  the  personnel  savings, 

GAO’s  analysis  fotmd  that  the  $24.2  million  of  persormel  savings  related  to 
the  Air  Force  and  the  Navy  were  not  supported. 

•  Air  Force  and  Navy  DTS  program  officials  stated  that  they  did  not 
anticipate  a  reduction  in  the  number  of  persormel,  but  rather  the  shifting 
of  staff  from  the  travel  function  to  other  functions. 

•  The  Naval  Cost  Analysis  Division  stated  that  the  Navy  will  not  realize  any 
tangible  personnel  cost  savings  from  the  implementation  of  DTS. 

In  regard  to  the  CTO  fees,  the  economic  analysis  assumed  that  70  percent  of 
all  DTS  airline  tickets  would  either  require  no  intervention  or  minimal 
intervention  from  the  CTOs,  resulting  in  an  estimated  armual  net  savings  of 
$31  million.  However,  the  sole  support  provided  by  the  DTS  program  office 
was  an  article  in  a  trade  industry  pubhcation.  The  article  was  not  based  on 
information  related  to  DTS,  but  rather  on  the  experience  of  one  private 
sector  company.  Furthermore,  the  economic  analysis  was  not  prepared  in 
accordance  with  guidance  prescribed  by  0MB  and  DOD. 

•  DOD  guidance  stated  that  the  life-cycle  cost  estimates  should  be  verified 
by  an  independent  party,  but  this  did  not  occm. 

•  The  economic  analysis  did  not  imdertake  an  assessment  of  the  effects  of 
the  uncertainty  inherent  in  the  estimates  of  benefits  and  costs.  Because 
an  economic  analysis  uses  estimates  and  assmnptions,  it  is  critical  that 
the  imprecision  in  both  the  imderlying  data  and  assumptions  be 
imderstood.  Such  an  assessment  is  referred  to  as  a  sensitivity  analysis. 

DOD  acknowledged  that  DTS  is  not  being  used  to  the  fullest  extent  possible, 
but  lacks  comprehensive  data  to  effectively  monitor  its  utilization.  DOD’s 
utilization  data  are  based  on  a  model  that  was  developed  in  calendar  year 
2003.  However,  the  model  has  not  been  completely  updated  to  reflect  actual 
DTS  usage.  The  lack  of  accmate  utilization  data  hinders  management’s 
ability  to  monitor  progress  toward  the  DOD  vision  of  DTS  as  the  standard 
travel  system.  GAO  also  foimd  that  the  military  services  have  initiated 
actions  that  are  aimed  at  increasing  the  utilization  of  DTS. 

Finally,  GAO  found  that  DTS  still  has  not  addressed  the  underlying  problems 
associated  with  weak  requirements  management  and  system  testing.  While 
DOD  has  acted  to  address  concerns  GAO  previously  raised,  GAO  found  that 
DTS’s  requirements  are  still  ambiguous  and  conflicting.  For  example,  DTS 
displaying  up  to  25  flights  for  each  inquiry  is  questionable  because  it  is 
imclear  whether  this  is  a  valid  requirement.  Until  DOD  improves  DTS’s 
requirements  management  practices,  the  department  will  not  have 
reasonable  assurance  that  DTS  can  provide  the  intended  functionality. 

_ United  States  Government  Accountability  Office 


Mr.  Chairman  and  Members  of  the  Subcommittee: 


Thank  you  for  the  opportunity  to  discuss  our  recent  report^  related  to 
problems  encountered  by  the  Department  of  Defense  (DOD)  in  its  efforts 
to  successfully  implement  the  Defense  Travel  System  (DTS).  As  you  know, 
DOD  envisions  DTS  as  the  department’s  standard  end-to-end  travel 
system.^  The  department  estimates  that  DTS  will  be  fully  deployed  at  all 
11,000  intended  locations  during  fiscal  year  2007.^  The  September  2003 
economic  analysis  noted  that  DTS,  when  fully  implemented,  would  result 
in  annual  net  savings  of  over  $56  million.  The  economic  analysis  noted  that 
savings  would  be  realized  by  the  department  during  fiscal  years  2009-2016. 
In  December  2003,  the  department’s  Chief  Information  Officer  approved  a 
DTS  funding  level  of  approximately  $564  million.  Of  this  amount,  the 
contract  for  the  design,  development,  and  deployment  of  DTS  was  for 
about  $264  million.  The  remaining  costs  are  associated  with  areas  such  as 
the  operation  and  maintenance  of  DTS,  operation  of  the  Program 
Management  Office-Defense  Travel  System  (PMO-DTS),  the  voucher 
payment  process,  and  management  and  oversight  of  the  numerous 
contracted  commercial  travel  offices  (CTO). 

My  testimony  today  is  based  on  our  September  2006  report,'*  which 
followed  up  on  our  September  2005  testimony  and  January  2006  report.** 
One  of  the  major  findings  in  our  previous  work  was  that  DOD  did  not  have 


*GAO,  Defense  Travel  System:  Reported  Savings  Questionable  and  Implementation 
Challenges  Remain,  GAO-06-980  (Washington,  D.C.:  Sept.  26,  2006). 

^DOD  expects  DTS  to  perform  all  functions  related  to  travel  or  ensure  that  other  systems 
are  provided  with  adequate  information  to  provide  this  functionality.  For  example, 
obligating  funds  associated  with  travel  is  a  necessary  function,  and  DTS  is  expected  to 
(1)  make  sure  that  adequate  funds  are  available  before  authorizing  travel  either  through 
information  contained  in  its  system  or  by  obtaining  the  necessary  information  from  another 
system,  (2)  obligate  funds  through  issuance  of  approved  travel  orders,  and  (3)  provide 
DOD’s  financial  management  systems  with  the  necesscuy  information  so  that  those  systems 
can  record  the  obligation.  Since  DTS  is  required  to  ensure  that  all  travel-related 
functionality  is  properly  performed,  DOD  commonly  refers  to  DTS  as  an  “end-to-end  travel 
system.” 

^As  of  September  2005,  the  department  had  estimated  that  DTS  would  be  fully  deployed 
during  fiscal  year  2006. 

^GAO-06-980. 

^GAO,  DOD  Business  Transformation:  Defense  Travel  System  Continues  to  Face 
Implementation  Challenges,  GAO-06-18  (Washington,  D.C.:  Jan.  18,  2006),  and  DOD 
Business  Transformation:  Preliminary  Observations  on  the  Defense  Travel  System, 
GAO-05-998T  (Washington,  D.C.:  Sept.  29,  2005). 
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reasonable  assurance  that  flight  information  was  properly  displayed  for 
DOD  travelers  because  the  department  failed  to  properly  test  the  system 
interfaces  through  which  the  data  are  accessed  for  display.  We  further 
noted  that  the  continued  use  of  the  existing  legacy  travel  systems  at 
locations  where  DTS  has  been  deployed  results  in  undemtilization  of  DTS 
and  reduces  the  envisioned  savings. 

Today,  1  will  highlight  three  key  findings  from  our  September  2006  report: 

•  Two  key  assumptions  related  to  the  estimated  cost  savings  in  the 
September  2003  DTS  economic  analysis  were  not  reasonable.  DOD 
strongly  objected  to  this  finding,  and  1  will  discuss  why  we  continue  to 
believe  that  our  finding  is  accurate. 

•  The  department  did  not  have  quantitative  metrics  to  measure  the  extent  to 
which  DTS  is  actually  being  used. 

•  DOD  still  has  not  addressed  several  functional  problems  associated  with 
weak  requirements  management  and  system  testing. 

Finally,  1  will  discuss  our  recommendations  to  improve  the  department’s 
management  and  oversight  of  DTS. 

Our  work  focused  on  the  validity  of  the  assumptions  that  were  the 
principal  drivers  of  DOD’s  net  annual  estimated  savings  of  over 
$56  million.  We  determined  that  the  data  were  sufficiently  reliable  for  this 
purpose.  We  did  not  review  the  accmacy  and  reliability  of  the  specific 
dollar  amounts  shown  in  the  September  2003  economic  analysis.  To 
address  our  objectives  we  also  (1)  reviewed  the  September  2003  economic 
analysis  and  met  with  cognizant  officials,  (2)  analyzed  DTS  utilization  data 
and  obtained  an  overview  of  the  method  and  data  used  by  DTS  program 
officials  to  report  the  rate  of  DTS  utilization  for  the  various  DOD 
components,  and  (3)  analyzed  DTS  flight  information.  Our  work  was 
performed  from  October  2005  through  July  2006  in  accordance  with  U.S. 
generally  accepted  government  auditing  standards.  Further  details  on  our 
scope  and  methodology  are  included  in  our  September  2006  report.® 


®GAO-06-980. 
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Summary 


Our  analysis  of  the  September  2003  DTS  economic  analysis  found  that  two 
key  assumptions  used  to  estimate  cost  savings  were  not  based  on  reliable 
information.  Two  primary  areas  represented  the  majority  of  the  over 
$56  million  of  estimated  annual  net  savings  DTS  was  expected  to  realize — 
personnel  savings  of  $24.2  million  and  reduced  CTO  fees  of  $31  million. 
The  $24.2  million  estimated  annual  personnel  savings  were  attributed  to 
the  Air  Force  and  Navy.^  However,  Air  Force  and  Navy  DTS  officials  stated 
that  they  did  not  anticipate  a  reduction  in  the  number  of  personnel  with 
the  full  implementation  of  DTS,  but  rather  the  shifting  of  staff  to  other 
functions.  Further,  the  Naval  Cost  Analysis  Division  has  stated  that  the 
Navy  will  not  realize  any  tangible  personnel  cost  savings  from  the 
implementation  of  DTS.  In  written  comments  on  a  draft  of  our  report,  the 
Under  Secretary  of  Defense  (Personnel  and  Readiness),  strongly  objected 
to  our  finding  that  the  estimated  personnel  savings  are  unrealistic  and 
stated  that  recognizing  fiscal  constraints,  the  department  continues  to 
identify  efficiencies  and  eliminate  redundancies  to  help  leverage  available 
funds.  As  noted  in  our  report,  DOD  officials  responsible  for  reviewing 
economic  analyses  stated  that  while  shifting  personnel  to  other  functions 
is  considered  a  benefit,  it  should  be  considered  an  intangible  benefit  rather 
than  tangible  dollar  savings  since  the  shifting  of  personnel  does  not  result 
in  a  reduction  of  DOD  expenditures.  Because  none  of  the  military  services 
could  validate  an  actual  reduction  in  the  number  of  personnel  as  a  result 
of  DTS  implementation,  and  DOD’s  comments  did  not  include  any 
additional  support  or  documentation  for  its  position,  we  continue  to 
believe  that  the  estimated  annual  personnel  savings  of  $54.1  million  are 
unrealistic. 

In  regard  to  the  estimated  annual  savings  of  $31  million  attributed  to  lower 
CTO  fees,  we  requested,  but  the  PMO-DTS  could  not  provide,  any  analysis 
of  travel  data  to  support  the  assumption  that  70  percent  of  all  airline 
tickets  would  be  considered  “no  touch” — meaning  that  there  would  be  no 
or  minimal  intervention  by  the  CTO,  thereby  resulting  in  lower  CTO  fees. 
We  found  that  the  70  percent  assumption  was  based  solely  upon  an  article 
that  appeared  in  a  travel  industry  trade  publication.  Further,  the  economic 
analysis  assumed  that  the  Navy  would  save  about  $7.5  million,  almost  25 
percent,  of  the  total  savings  related  to  CTO  fees  once  DTS  is  fully 
deployed.  Again,  this  figure  was  based  on  a  reduction  in  the  fees  the  Navy 
would  pay  for  “no  touch”  transactions.  However,  the  Navy  paid  a  flat 


^The  economic  analysis  identified  annual  savings  of  $11.3  million  and  $12.9  million  for  the 
Air  Force  and  Navy,  respectively. 
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management  fee  that  was  the  same  regardless  of  the  involvement  of  the 
CTO — therefore,  the  reduced  “no  touch”  fee  would  not  apply. 

In  addition,  the  economic  analysis  was  not  prepared  in  accordance  with 
guidance  prescribed  by  the  Office  of  Management  and  Budget  (0MB)  and 
DOD.  Both  sets  of  guidance  require  that  an  economic  analysis  be  based  on 
facts  and  data  and  be  explicit  about  the  underlying  assumptions  used  to 
arrive  at  future  benefits  and  costs.  DOD  guidance  also  states  that  life-cycle 
cost  estimates  should  be  independently  validated.  An  independent  review 
is  intended,  in  part,  to  provide  program  management  some  degree  of 
assurance  that  the  life-cycle  cost  estimates  are  reasonable  and  the  cost 
estimates  are  built  on  realistic  program  assumptions.  However,  an 
independent  validation  was  not  performed. 

Our  analysis  also  found  that  the  department  did  not  have  quantitative 
metrics  to  measure  the  extent  to  which  DTS  is  actually  being  used.  The 
reported  DTS  utilization  rates  were  based  on  a  methodology  that  was 
developed  using  estimated  data,  and  PMO-DTS  program  officials 
acknowledged  that  the  model  had  not  been  completely  updated  with 
actual  data  as  DTS  continued  to  be  implemented  at  the  11,000  sites.  As  a 
result,  the  PMO-DTS  continues  to  rely  on  outdated  irrformation  in 
calculating  DTS  utilization  rates  that  are  reported  to  DOD  management 
and  the  Congress.  Additionally,  while  the  military  services  have  initiated 
actions  to  help  increase  the  utilization  of  DTS,  they  pointed  out  that 
ineffective  DTS  training  is  a  contributing  factor  to  the  lower  than  expected 
usage  rate  by  the  military  services. 

Finally,  DOD  still  has  not  addressed  several  functional  problems 
associated  with  weak  requirements  management  and  system  testing. 
Requirements  represent  the  blueprint  that  system  developers  and  program 
managers  use  to  design,  develop,  test,  and  implement  a  system.  Because 
requirements  provide  the  foundation  for  system  testing,  they  must  be 
complete,  clear,  and  well  documented  to  design  and  implement  an 
effective  testing  program.  Adequately  defined  and  tested  requirements  are 
one  of  the  key  elements  to  help  reduce  a  project’s  risks  to  acceptable 
levels.®  Our  February  2006  analysis  disclosed  that  DOD  still  did  not  have 


^Acceptable  levels  refer  to  the  fact  that  any  systems  acquisition  effort  will  have  risks  and 
will  suffer  the  adverse  consequences  associated  with  defects  in  the  processes.  However, 
effective  implementation  of  disciplined  processes,  which  includes  project  planning  and 
management,  requirements  management,  risk  management,  quality  assurance,  and  testing, 
reduces  the  possibility  of  the  potential  risks  actually  occurring  and  prevents  significant 
defects  from  materially  affecting  the  cost,  timeliness,  and  performance  of  the  project. 
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reasonable  assurance  that  the  flight  information  was  being  properly 
displayed  to  DOD  travelers.  We  identified  246  unique  GSA  city  pair  flights 
that  should  have  been  identified  on  one  or  more  DTS  flight  displays 
according  to  the  DOD  requirements.  However,  87  of  these  flights  did  not 
appear  on  one  or  more  of  the  required  listings.  While  the  PMO-DTS  has 
taken  action  to  address  our  concerns,  these  actions  do  not  fully  address 
the  fundamental  problems  we  found  during  this  audit  and  on  which  we 
have  previously  reported.'*  For  example,  the  DTS  requirements  we 
reviewed  were  still  ambiguous  and  conflicting. 

Our  September  2006  report  includes  four  recommendations  to  the 
Secretary  of  Defense  aimed  at  improving  the  department’s  management 
and  oversight  of  DTS.  We  recommended  that  the  Secretary  of  Defense 
(1)  evaluate  the  cost  effectiveness  of  the  Navy  continuing  with  the  CTO 
management  fee  structure,  (2)  update  the  DTS  Voucher  Analysis  Model  to 
report  DTS  actual  utilization  rates,  (3)  require  the  PMO-DTS  to  provide 
periodic  reports  on  the  utilization  of  DTS,  and  (4)  resolve  inconsistencies 
in  DTS  requirements.  DOD  generally  agreed  with  the  recommendations 
and  described  its  efforts  to  address  them. 


Validity  of  DTS 
Economic  Analysis 
Questionable 


In  September  2003,  DOD  finalized  its  economic  analysis  for  DTS  in 
preparation  for  a  milestone  decision  review.*"  The  highlights  of  the 
economic  analysis  are  shown  in  table  1.  In  December  2003,  the  DOD  Chief 
Irrformation  Officer  granted  approval  for  DTS  to  proceed  with  full 
implementation  throughout  the  department. 


"GAO-05-998T  and  GAO-06-18. 

*°The  September  2003  economic  analysis  is  an  addendum  to  the  July  2003  DTS  economic 
analysis. 


Page  5 


GAO-07-208T 


Table  1 :  Summary  of  DTS  Estimated  Annual  Net  Savings  Reported  in  the  September 
2003  Economic  Analysis 

Constant  fiscal  year  2003  dollars  in  millions 

Cost  components 

Estimated  annual  net  savings 

Records  management 

$19.8 

Centrally  billed  accounts 

1.7 

CTO  acquisition  and  administration 

2.4 

CTO  services 

31.0 

Voucher  process  and  compute 

54.1 

Voucher  pay 

0 

Legacy  systems 

14.5 

PMO 

(8.8) 

Help  desk/DTA 

(36.8) 

System  operations 

(21.5) 

Total  net  savings 

$56.4 

Source:  September  2003  economic  analysis  provided  by  the  PMO-DTS. 


Note:  In  arriving  at  the  estimated  annual  net  savings  of  over  $56  million,  the  economic  analysis  took 
into  consideration  the  estimated  costs  of  over  $2.1  billion,  which  covers  fiscal  years  2003-2016.  The 
estimated  costs  included  the  costs  that  are  estimated  to  be  incurred  by  the  PMO-DTS,  the  Army,  the 
Navy,  the  Air  Force,  and  the  defense  agencies. 

Our  analysis  of  the  September  2003  DTS  economic  analysis  found  that  two 
key  assumptions  used  to  estimate  cost  savings  were  not  based  on  reliable 
information.  Consequently,  the  economic  analysis  did  not  serve  to  help 
ensure  that  the  funds  invested  in  DTS  were  used  in  an  efficient  and 
effective  manner.  Two  primary  areas — ^persormel  savings  and  reduced 
CTO  fees — represented  the  majority  of  the  over  $56  million  of  estimated 
annual  net  savings  DTS  was  expected  to  realize.  However,  the  estimates 
used  to  generate  these  savings  were  unreliable.  Further,  DOD  did  not 
effectively  implement  the  policies  relating  to  developing  economic 
analyses  for  programs  such  as  DTS.  Effective  implementation  of  these 
policies  should  have  highlighted  the  problems  that  we  found  and  allowed 
for  appropriate  adjustments  so  that  the  economic  analysis  could  have 
served  as  a  useful  management  tool  in  making  funding  decisions  related  to 
DTS — which  is  the  primary  purpose  of  this  analysis.  While  the 
department’s  system  acquisition  criteria  do  not  require  that  a  new 
economic  analysis  be  prepared,  the  department’s  business  system 
investment  management  structure  provides  an  opportunity  for  DOD 
management  to  assess  whether  DTS  is  meeting  its  planned  cost,  schedule, 
and  functionality  goals. 
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Personnel  Savings  Are 
Unrealistic 


The  economic  analysis  estimated  that  the  annual  personnel  savings  was 
over  $64  million, “  as  shown  in  table  2. 


Table  2:  Summary  of  Estimated  Annual  Personnel  Savings 

Constant  fiscal  year  2003  dollars  In  millions 

DOD  component 

Estimated  annual  savings 

Army 

$16.0 

Navy 

12.9 

Air  Force 

11.3 

Marine  Corps 

5.8 

Defense  agencies 

6.3 

Permanent  change  of  station 

1.8 

Total  savings 

$54.1 

Source:  September  2003  economic  analysis  provided  by  the  PMO-DTS. 


As  shown  in  table  2,  approximately  45  percent  of  the  estimated  savings,  or 
$24.2  million,  was  attributable  to  the  Air  Force  and  Navy.  The  assumption 
behind  the  personnel  savings  computation  was  that  there  would  be  less 
manual  intervention  in  the  processing  of  travel  vouchers  for  payment,  and 
therefore  fewer  staff  would  be  needed.  However,  based  on  our  discussions 
with  Air  Force  and  Navy  DTS  program  officials,  it  is  questionable  as  to 
how  the  estimated  savings  will  be  achieved.  Air  Force  and  Navy  DTS 
program  officials  stated  that  they  did  not  anticipate  a  reduction  in  the 
number  of  personnel  with  the  full  implementation  of  DTS,  but  rather  the 
shifting  of  staff  to  other  functions.  According  to  DOD  officials  responsible 
for  reviewing  economic  analyses,  while  shifting  personnel  to  other 
functions  is  considered  a  benefit,  it  should  be  considered  an  intangible 
benefit  rather  than  tangible  dollar  savings  since  the  shifting  of  personnel 
does  not  result  in  a  reduction  of  DOD  expenditures.  Also,  as  part  of  the 
Navy’s  overall  evaluation  of  the  economic  analysis,  program  officials 
stated  that  “the  Navy  has  not  identified,  and  conceivably  will  not 
recommend,  any  personnel  billets  for  reduction.”  Finally,  the  Naval  Cost 
Analysis  Division  (NCAD)  October  2003  report  on  the  economic  analysis 
noted  that  it  could  not  validate  approximately  40  percent  of  the  Navy’s 
total  costs,  including  personnel  costs,  in  the  DTS  life-cycle  cost  estimates 
because  credible  supporting  documentation  was  lacking.  The  report  also 


^'During  fiscal  years  2009  through  2016. 
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noted  that  the  PMO-DTS  used  unsound  methodologies  in  preparing  the 
DTS  economic  analysis. 


The  extent  of  personnel  savings  for  the  Army  and  defense  agencies,  which 
are  reported  as  $16  million  and  $6.3  million  respectively,  is  also  unclear. 
The  Army  and  many  defense  agencies  use  the  Defense  Finance  and 
Accounting  Service  (DFAS)  to  process  their  travel  vouchers,  so  the 
personnel  savings  for  the  Army  and  the  defense  agencies  were  primarily 
related  to  reductions  in  DFAS’s  costs.  In  discussions  with  DFAS  officials, 
they  were  unable  to  estimate  the  actual  personnel  savings  that  would 
result  since  they  did  not  know  (1)  the  number  of  personnel,  like  those  at 
the  Air  Force  and  Navy,  that  would  simply  be  transferred  to  other  DFAS 
functions  or  (2)  the  number  of  personnel  that  could  be  used  to  avoid 
additional  hiring.  For  example,  DFAS  expects  that  some  of  the  individuals 
assigned  to  support  the  travel  function  could  be  moved  to  support  its 
ePayroll  program.  Since  these  positions  would  need  to  be  filled  regardless 
of  whether  the  travel  function  is  reduced,  transferring  personnel  from 
travel  to  ePayroll  would  reduce  DOD’s  overall  costs  since  DFAS  would  not 
have  to  hire  additional  individuals. 

DOD  strongly  objected  to  our  finding  that  the  personnel  savings  are 
unrealistic.  In  its  written  comments,  the  department  stated  that  it  is  facing 
an  enormous  challenge  and  continues  to  identify  efficiencies  and  eliminate 
redundancies  to  help  leverage  available  funds.  We  fully  recognize  that  the 
department  is  attempting  to  improve  the  efficiency  and  effectiveness  of  its 
business  operations.  The  Comptroller  General  of  the  United  States 
testified  in  August  2006  that  increased  commitment  by  the  department  to 
address  DOD’s  numerous  challenges  represents  an  improvement  over  past 
efforts. 

The  fact  remains,  however,  that  the  results  of  an  economic  analysis  are 
intended  to  help  management  decide  if  future  investments  in  a  given 
endeavor  are  worthwhile.  In  order  to  provide  management  with  this 
information,  it  is  imperative  that  the  underlying  assumptions  in  an 
economic  analysis  be  supported  by  valid  assumptions.  The  September 
2003  economic  analysis  noted  that  persormel  savings  of  $54.1  million 
would  be  realized  by  the  department  annually  for  fiscal  years  2009  through 


^^GAO,  Department  of  Defense:  Sustained  Leadership  Is  Critical  to  Effective  Financial 
and  Business  Management  Transformation,  GAO-06-1006T  (Washington,  D.C.:  Aug.  3, 
2006). 
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Savings  Associated  with 
Reduction  of  CTO  Fees 
Are  Unknown 


“No  Touch”  Transaction 
Volume  Estimates  Are  Not 
Supported 


2016.  However,  based  on  our  review  and  analysis  of  documentation  and 
discussion  with  department  personnel,  we  found  that  the  underlying 
assumptions  in  support  of  the  $54.1  million  were  not  valid,  particularly  in 
regard  to  the  amounts  estimated  for  the  Navy  and  Air  Force.  For  example, 
we  agree  with  the  statements  of  DOD  officials  who  indicated  that  the 
shifting  of  persormel  to  other  functions  cannot  be  counted  towards 
tangible  dollar  savings,  since  such  actions  do  not  result  in  a  reduction  of 
DOD  expenditures.  Moreover,  the  department  did  not  provide  any  new 
data  or  related  documentation  in  its  comments  that  were  counter  to  our 
finding.  As  a  result  of  these  factors,  we  continue  to  believe  that  the 
estimated  annual  personnel  savings  of  $54.1  million  are  unrealistic. 


According  to  the  September  2003  economic  analysis,  DOD  expected  to 
realize  annual  net  savings  of  $31  million  through  reduced  fees  paid  to  the 
CTOs  because  the  successful  implementation  of  DTS  would  enable  the 
majority  of  airline  tickets  to  be  acquired  with  either  no  or  minimal 
intervention  by  the  CTOs.  These  are  commonly  referred  to  as  “no  touch” 
transactions.  However,  DOD  did  not  have  a  sufficient  basis  to  estimate  the 
number  of  transactions  that  would  be  considered  “no  touch”  since  the 
(1)  estimated  percentage  of  transactions  that  can  be  processed  using  the 
“no  touch”  was  not  supported  and  (2)  analysis  did  not  properly  consider 
the  effects  of  components  that  use  management  fees,  rather  than 
transaction  fees,  to  compensate  the  CTOs  for  services  provided.  The 
weaknesses  we  identified  with  the  estimating  process  raise  serious 
questions  as  to  whether  DOD  will  realize  substantial  portions  of  the 
estimated  annual  net  savings  of  $31  million. 

DOD  arrived  at  the  $31  million  of  annual  savings  in  CTO  fees  by  estimating 
that  70  percent  of  all  DTS  airline  tickets  would  be  considered  “no  touch” 
and  then  multiplying  these  tickets  by  the  savings  per  ticket  in  CTO  fees. 
However,  a  fundamental  flaw  in  this  analysis  was  that  the  70  percent 
assumption  had  no  solid  basis.  We  requested,  but  the  PMO-DTS  could  not 
provide,  any  analysis  of  travel  data  to  support  the  assertion.  Rather,  the 
sole  support  provided  by  the  PMO-DTS  was  an  article  in  a  travel  industry 
trade  publication.  “  The  article  was  not  based  on  irrformation  related  to 
DTS,  but  rather  on  the  experience  of  one  private  sector  company. 


'^American  Express  News  Releases:  American  Express’  Interactive  Travel  Update,  (New 
York,  N.Y.:  Aug.  11,  2003),  http://corp.americanexpress.coiu/gcs/cards/us/ni/pr/081303.aspx. 
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The  economic  analysis  assumed  that  DOD  could  save  about  $13.50  per  “no 
touch”  ticket.  Since  that  analysis,  DOD  has  awarded  one  contract  that 
specifically  prices  transactions  using  the  same  model  as  that  envisioned  by 
the  economic  analysis.  This  contract  applies  to  the  Defense  Travel  Region 
6  travel  area.''*  During  calendar  year  2005,  the  difference  in  fees  for  “no 
touch”  transactions  and  the  transactions  supported  by  the  current  process 
averaged  between  $10  and  $12,  depending  on  when  the  fees  were  incurred 
because  the  contract  rates  changed  during  2005.“^  In  analyzing  travel 
voucher  data  for  Region  6  for  calendar  year  2005,  we  found  that  the 
reported  “no  touch”  rate  was,  at  best  47  percent — ^far  less  than  the  70 
percent  envisioned  in  the  economic  analysis. 

PMO-DTS  program  officials  stated  they  are  uncertain  as  to  why  the 
anticipated  70  percent  “no  touch”  was  not  being  achieved.  According  to 
PMO-DTS  program  officials,  this  could  be  attributed,  in  part,  to  the  DOD 
travelers  being  uncomfortable  with  the  system  and  with  making 
reservations  without  using  a  CTO.  Although  this  may  be  one  reason,  other 
factors  may  also  affect  the  expected  “no  touch”  fee.  For  example,  we  were 
informed  that  determining  the  airline  availability  and  making  the 
associated  reservation  can  be  accomplished,  in  most  cases,  rather  easily. 
However,  obtaining  information  related  to  hotels  and  rental  cars  and 
making  the  associated  reservation  can  be  more  problematic  because  of  the 
limitations  in  the  data  that  DTS  is  able  to  obtain  from  its  commercial 
sources.  Accordingly,  while  a  traveler  may  be  able  to  make  a  “no  touch” 
reservation  for  the  airline  portion  of  the  trip,  the  individual  may  need  to 
contact  the  CTO  in  order  to  make  hotel  or  rental  car  reservations.  When 
this  occurs,  rather  than  paying  a  “no  touch”  fee  to  the  CTO,  DOD  ends  up 
paying  a  higher  fee,  which  eliminates  the  savings  estimated  in  the 
economic  analysis. 


^^Defense  Travel  Region  6  includes  the  Air  Force  and  defense  agencies  in  the  states  of 
Kentucky,  Illinois,  Indiana,  Iowa,  Michigan,  Minnesota,  Missouri,  Nebraska,  North  Dakota, 
South  Dakota,  and  Wisconsin.  The  contract  also  applies  to  Army  activities  in  8  of  the  11 
states  (excluding  Kentucky,  Missouri,  and  Nebraska).  As  discussed  later,  the  Navy  uses  a 
management  fee  contract,  and  is  therefore  not  included  in  the  Defense  Travel  Region  6 
contract. 

^According  to  DTS  officials,  these  savings  are  consistent  with  the  DTS  contracts  that  have 
been  awarded  to  small  businesses.  The  average  savings  per  “no  touch”  ticket  under  these 
contracts  is  about  $12.88.  Because  the  contractors  are  paid  these  fees  directly  by  the 
traveler,  they  are  unable  to  determine  the  percentage  of  transactions  that  are  actually  paid 
using  the  “no  touch”  rate. 
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Navy  Impact  of  CTO 
Management  Fees  Not 
Adequately  Considered 


The  economic  analysis  assumed  that  (1)  DOD  would  be  able  to  modify  the 
existing  CTO  contracts  to  achieve  a  substantial  reduction  in  fees  paid  to  a 
CTO  when  DTS  was  fully  implemented  across  the  department  and  (2)  all 
services  would  use  the  fee  structure  called  for  in  the  new  CTO  contracts. 
The  first  part  of  the  assumption  is  supported  by  results  of  the  CTO 
contract  for  DOD  Region  6  travel.  The  fees  for  the  DTS  “no  touch” 
transactions  were  at  least  $10  less  than  if  a  CTO  was  involved  in  the 
transactions.  However,  to  date,  the  department  has  experienced  difficulty 
in  awarding  new  contracts  with  the  lower  fee  structure.  On  May  10,  2006, 
the  department  announced  the  cancellation  of  the  solicitation  for  a  new 
contract.  According  to  the  department,  it  decided  that  the  solicitation 
needed  to  be  rewritten  based  on  feedback  from  travel  industry 
representatives  at  a  March  28,  2006,  conference.  The  department 
acknowledged  that  the  “DTS  office  realized  its  solicitation  didn’t  reflect 
what  travel  agency  services  it  actually  needed.”'®  The  department  would 
not  say  how  the  solicitation  would  be  refined,  citing  the  sensitivity  of  the 
procurement  process.  The  department  also  noted  that  the  new  solicitation 
would  be  released  soon,  but  provided  no  specific  date. 

The  economic  analysis  assumed  that  the  Navy  would  save  about  $7.5 
million,  almost  25  percent,  of  the  total  savings  related  to  CTO  fees  once 
DTS  is  fully  deployed.  The  economic  analysis  averaged  the  CTO  fees  paid 
by  the  Army,  the  Air  Force,  and  the  Marine  Corps — ^which  amounted  to 
about  $18.71  per  transaction — to  compute  the  savings  in  Navy  CTO  fees. 
Using  these  data,  the  assumption  was  made  in  the  economic  analysis  that 
a  fee  of  $5.25  would  be  assessed  for  each  ticket,  resulting  in  an  average 
savings  of  $13.46  per  ticket  for  the  Navy  ($18.71  minus  $5.25).'^  While  this 
approach  may  be  valid  for  the  organizations  that  pay  individual  CTO  fees, 
it  may  not  be  representative  for  organizations  such  as  the  Navy  that  pay  a 
management  fee.  The  management  fee  charged  the  Navy  is  the  same 
regardless  of  the  involvement  of  the  CTO — therefore,  the  reduced  “no 
touch”  fee  would  not  apply. 

We  were  informed  by  Navy  DTS  program  officials  that  they  were 
considering  continuing  the  use  of  management  fees  after  DTS  is  fully 
implemented.  According  to  Navy  DTS  program  officials,  they  paid  about 
$14.5  million  during  fiscal  year  2005  for  CTO  management  fees,  almost  $19 


'®“DOD  Retracts  Solicitation  for  Travel  Agency  Services,”  FederalTimes.com  (May  16, 
2006),  http://www.federaltimes.com/index.php?  (downloaded  June  14,  2006). 

"^These  savings  translate  to  about  572,000  tickets  annually. 
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per  ticket  for  approximately  762,700  tickets  issued.  Accordingly,  even  if 
the  department  arrives  at  a  new  CTO  contract  containing  the  new  fee 
structure  or  fees  similar  to  those  of  Region  6,  the  estimated  savings  related 
to  CTO  fees  for  the  Navy  will  not  be  realized  if  the  Navy  continues  to  use 
the  management  fee  concept. 


Effective  Implementation 
of  Existing  Policies  Should 
Have  Identified  Problems 
with  the  Economic 
Analysis 


Effective  implementation  of  DOD  guidance  would  have  detected  the  types 
of  problems  discussed  above  and  resulted  in  an  economic  analysis  that 
would  have  accomplished  the  stated  objective  of  the  process — to  help 
ensure  that  the  funds  invested  in  DTS  were  used  efficiently  and  effectively. 
DOD  policy^®  and  0MB  guidance^  require  that  an  economic  analysis  be 
based  on  facts  and  data  and  be  explicit  about  the  underlying  assumptions 
used  to  arrive  at  estimates  of  future  benefits  and  costs.  Since  an  economic 
analysis  deals  with  costs  and  benefits  occurring  in  the  future,  assumptions 
must  be  made  to  account  for  uncertainties.  DOD  policy  recognizes  this  and 
provides  a  systematic  approach  to  the  problem  of  choosing  the  best 
method  of  allocating  scarce  resources  to  achieve  a  given  objective. 


A  sound  economic  analysis  recognizes  that  there  are  alternative  ways  to 
meet  a  given  objective  and  that  each  alternative  requires  certain  resources 
and  produces  certain  results.  The  purpose  of  the  economic  analysis  is  to 
give  the  decision  maker  insight  into  economic  factors  bearing  on 
accomplishing  the  objectives.  Therefore,  it  is  important  to  identify  factors, 
such  as  cost  and  performance  risks  and  drivers,  that  can  be  used  to 
establish  and  defend  priorities  and  resource  allocations.  The  DTS 
economic  analysis  did  not  comply  with  the  DOD  policy,  and  the 
weaknesses  we  found  should  have  been  detected  had  the  DOD  policy  been 
effectively  implemented.  The  PMO-DTS  had  adequate  warning  signs  of  the 
potential  problems  associated  with  not  following  the  0MB  and  DOD 
guidance  for  developing  an  effective  economic  analysis.  For  example,  as 
noted  earlier,  the  Air  Force  and  Navy  provided  comments  when  the 
economic  analysis  was  being  developed  that  the  expected  benefits  being 
claimed  were  unrealistic.  Just  removing  the  benefits  associated  with 
personnel  savings  from  the  Air  Force  and  Navy  would  have  reduced  the 
overall  estimated  program  cost  savings  by  almost  45  percent.  This  would 
have  put  increased  pressure  on  the  credibility  of  using  a  70  percent  “no 
touch”  utilization  rate. 


^®DOD  Instruction  Economic  Analysis  for  Decisionmaking,  November  7,  1995. 

^^Office  of  Management  and  Budget,  Circular  No.  A-94,  Guidelines  and  Discount  Rates  for 
Benefit-Cost  Analysis  of  Federal  Programs  (Revised  Jan.  18,  2006). 
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Specific  examples  of  failures  to  effectively  implement  the  DOD  policy  on 
conducting  economic  analyses  include  the  (1)  DTS  life-cycle  cost 
estimates  portion  of  the  economic  analysis  was  not  independently 
validated  as  specified  in  DOD’s  guidance^”  and  (2)  September  2003  DTS 
economic  analysis  did  not  undertake  an  assessment  of  the  effects  of  the 
uncertainty  inherent  in  the  estimates  of  benefits  and  costs,  as  required  by 
DOD  and  0MB  guidance.^^  Because  an  economic  analysis  uses  estimates 
and  assumptions,  it  is  critical  that  a  sensitivity  analysis^^  be  performed  to 
understand  the  effects  of  the  imprecision  in  both  underlying  data  and 
modeling  assumptions. 


DTS  Remains 
Underutilized  by  the 
Military  Services 


Our  September  2005  testimony  and  January  2006  report^^  noted  the 
challenge  facing  the  department  in  attaining  the  anticipated  DTS 
utilization.  While  DOD  has  acknowledged  the  underutilization,  we  found 
that,  across  DOD,  the  department  does  not  have  reasonable  quantitative 
metrics  to  measure  the  extent  to  which  DTS  is  actually  being  used. 
Presently,  the  reported  DTS  utilization  is  based  on  a  DTS  Voucher  Analysis 
ModeP  that  was  developed  in  calendar  year  2003  using  estimated  data,  but 
over  the  years  has  not  been  completely  updated  with  actual  data.  While 
the  military  services  have  initiated  actions  to  help  increase  the  utilization 
of  DTS,  they  pointed  out  that  ineffective  DTS  training  is  a  contributing 
factor  to  the  lower  than  expected  usage  rate  by  the  military  services. 


^"Department  of  Defense  Instruction  5000.2,  Operation  of  the  Defense  Acquisition  System, 
May  12,  2003. 

^'Department  of  Defense  Instruction  7041.3,  Economic  Analysis  of  Decisionmaking,  (Nov. 
7,  1995),  and  Office  of  Management  and  Budget  Revised  Circular  No.  A-94,  Guidelines  and 
Discount  Rates  for  Benefit-Cost  Analysis  of  Federal  Programs  (Oct.  29,  1992). 

^^Sensitivity  analysis  refers  to  changing  the  value  of  a  given  variable  in  a  model  to  gauge  the 
effect  of  change  on  model  results. 

''^GAO-05-998T  and  GAO-06-18. 

^'DOD  developed  a  model  in  calendar  year  2003  that  compares  the  expected  usage  against 
the  actual  usage.  The  expected  usage  is  obtained  by  using  historical  data,  such  as  ticket 
counts,  to  determine  the  expected  number  of  vouchers  processed  by  a  given  location.  For 
example,  if  a  location  had  1,000  vouchers  as  its  expected  number  of  vouchers  per  the 
model,  but  now  processes  750  actual  vouchers  through  DTS,  then  the  PMO  model 
considers  that  that  location  has  achieved  a  75  percent  utilization  rate.  It  then  takes  the 
individual  computations  for  each  DTS  location  and  “rolls  them  up”  to  determine  the  total 
utilization  for  individual  service  performance  on  a  monthly  basis. 
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Metrics  to  Measure  DTS 
Utilization  Are  Inadequate 


The  DTS  Voucher  Analysis  Model  was  prepared  in  calendar  year  2003  and 
based  on  airline  ticket  and  voucher  count  data  that  were  reported  by  the 
military  services  and  defense  agencies,  but  the  data  were  not  verified  or 
validated.  Furthermore,  PMO-DTS  officials  acknowledged  that  the  model 
has  not  been  completely  updated  with  actual  data  as  DTS  continues  to  be 
implemented  at  the  11,000  sites.  We  found  that  the  Air  Force  is  the  only 
military  service  that  submits  monthly  metrics  to  the  PMO-DTS  officials  for 
their  use  in  updating  the  DTS  Voucher  Analysis  Model.  Rather  than 
reporting  utilization  based  on  individual  site  system  utilization  data,  the 
PMO-DTS  continues  to  rely  on  outdated  information  in  the  reporting  of 
DTS  utilization  to  DOD  management  and  the  Congress.  We  have 
previously  reported^^  that  best  business  practices  indicate  that  a  key  factor 
of  project  management  and  oversight  is  the  ability  to  effectively  monitor 
and  evaluate  a  project’s  actual  performance  against  what  was  planned. 

In  order  to  perform  this  critical  task,  best  business  practices  require  the 
adoption  of  quantitative  metrics  to  help  measure  the  effectiveness  of  a 
business  system  implementation  and  to  continually  measure  and  monitor 
results,  such  as  system  utilization.  This  lack  of  accurate  and  pertinent 
utilization  data  hinders  management’s  ability  to  monitor  its  progress 
toward  the  DOD  vision  of  DTS  as  the  standard  travel  system,  as  well  as  to 
provide  consistent  and  accurate  data  to  Congress.  With  the  shift  of  the 
DTS  program  to  the  Business  Transformation  Agency  (BTA),^®  which  now 
makes  DTS  an  enterprisewide  endeavor,  improved  metrics  and  training 
are  essential  if  DTS  is  to  be  DOD’s  standard,  integrated,  end-to-end  travel 
system  for  business  travel. 

DTS’s  reported  utilization  rates  for  the  period  October  2005  through  April 
2006  averaged  53  percent  for  Army,  30  percent  for  Navy,  and  39  percent 
for  Air  Force.  Because  the  PMO-DTS  was  not  able  to  identify  the  total 
number  of  travel  vouchers  that  should  have  been  processed  through  DTS 


^^GAO,  Financial  Management  Systems:  Additional  Efforts  Needed  to  Address  Key 
Causes  of  Modernization  Failures,  GAO-06-184  (Washington,  D.C.:  Mar.  15,  2006),  and 
Financial  Management  Systems:  Lack  of  Disciplined  Processes  Puts  Implementation  of 
HHS’  Financial  System  at  Risk,  GAO-04-1008  (Washington,  D.C.:  Sept.  23,  2004). 

^®In  October  2005,  DOD  established  BTA  to  advance  DOD-wide  business  transformation 
efforts,  particularly  with  regard  to  business  systems  modernization.  BTA  operates  under 
the  authority,  direction,  and  control  of  the  Under  Secretary  of  Defense  for  Acquisition, 
Technology,  and  Logistics,  who  is  the  vice  chair  of  the  Defense  Business  Systems 
Management  Committee — which  serves  as  the  highest  ranking  governing  body  for  business 
systems  modernization  activities. 
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(total  universe  of  travel  vouchers),  these  utilization  rates  may  be  over-  or 
understated.  PMO-DTS  program  officials  confirmed  that  the  reported 
utilization  data  were  not  based  on  complete  data  because  the  department 
did  not  have  comprehensive  information  to  identify  the  universe  or  the 
total  number  of  travel  vouchers  that  should  be  processed  through  DTS. 
PMO-DTS  program  and  DTS  military  service  officials  agreed  that  the 
actual  DTS  utilization  rate  should  be  calculated  by  comparing  actual 
vouchers  being  processed  in  DTS  to  the  total  universe  of  vouchers  that 
should  be  processed  in  DTS.  The  universe  would  exclude  those  travel 
vouchers  that  cannot  be  processed  through  DTS,  such  as  those  related  to 
permanent  change  of  station  travel. 

The  Air  Force  was  the  only  military  service  that  attempted  to  obtain  data 
on  (1)  the  actual  travel  vouchers  processed  through  DTS  and  (2)  those 
travel  vouchers  that  were  eligible  to  be  processed  through  DTS,  but  were 
not.  These  data  were  site-specific.  For  example,  during  the  month  of 
December  2005,  the  PMO-DTS  reported  that  at  Wright-Patterson  Air  Force 
Base,  2,880  travel  vouchers  were  processed  by  DTS,  and  the  Air  Force 
reported  that  another  2,307  vouchers  were  processed  through  the  legacy 
system — the  Reserve  Travel  System  (RTS).  Of  those  processed  through 
RTS,  Air  Force  DTS  program  officials  stated  that  338  travel  vouchers 
should  have  been  processed  through  DTS.  DTS  Air  Force  program  officials 
further  stated  that  they  submitted  to  the  PMO-DTS  the  number  of  travel 
vouchers  processed  through  RTS  each  month.  These  data  are  used  by  the 
PMO-DTS  to  update  the  DTS  Voucher  Analysis  Model.  However,  neither 
the  Air  Force  nor  the  PMO-DTS  have  verified  the  accuracy  and  reliability 
of  the  data.  Therefore,  the  accuracy  of  the  utilization  rates  reported  for  the 
Air  Force  by  the  PMO-DTS  is  not  known. 

Because  Army  and  Navy  DTS  program  officials  did  not  have  the 
information  to  identify  the  travel  transactions  that  should  have  been 
processed  through  DTS,  the  Army  and  Navy  did  not  have  a  basis  for 
evaluating  DTS  utilization  at  their  respective  military  locations  and 
activities.  Furthermore,  Navy  DTS  program  officials  indicated  that  the 
utilization  data  that  the  PMO-DTS  program  officials  reported  for  the  Navy 
were  not  accurate.  According  to  Navy  DTS  program  officials,  the  Navy’s 
primary  source  of  utilization  data  was  the  monthly  metrics  reports 
provided  by  the  PMO-DTS,  but  Navy  DTS  program  officials  questioned  the 
accuracy  of  the  Navy  utilization  reports  provided  by  the  PMO-DTS. 
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DOD  Has  Taken  Steps  to 
Improve  DTS  Utilization, 
but  Further  Action  Is 
Needed 


Although  the  military  services  have  issued  various  memorandums  aimed  at 
increasing  the  utilization  of  DTS,  the  military  service  DTS  program 
officials  all  pointed  to  ineffective  training  as  a  primary  cause  of  DTS  not 
being  utilized  to  a  far  greater  extent.  The  following  examples  highlight  the 
concerns  raised  by  the  military  service  officials: 


•  Army  DTS  program  officials  emphasized  that  the  DTS  system  is  complex 
and  the  design  presents  usability  challenges  for  users — especially  for  first¬ 
time  or  infrequent  users.  They  added  that  a  major  concern  is  that  there  is 
no  PMO-DTS  training  for  existing  DTS  users  as  new  functionality  is  added 
to  DTS.  These  officials  stated  that  the  PMO-DTS  does  not  do  a  good  job  of 
informing  users  about  functionality  changes  made  to  the  system.  We 
inquired  if  the  Help  Desk  was  able  to  resolve  the  users’  problems,  and  the 
Army  DTS  officials  simply  stated  “no.”  The  Army  officials  further  pointed 
out  that  it  would  be  beneficial  if  the  PMO-DTS  improved  the  electronic 
training  on  the  DTS  Web  site  and  made  the  training  documentation  easier 
to  understand.  Also,  improved  training  would  help  infrequent  users  adapt 
to  system  changes.  The  Army  officials  noted  that  without  some  of  these 
improvements  to  resolve  usability  concerns,  DTS  will  continue  to  be 
extremely  frustrating  and  cumbersome  for  travelers. 


•  Navy  DTS  program  officials  stated  that  DTS  lacks  adequate  user/traveler 
training.  The  train-the-trainer  concept  of  training  system  administrators 
who  could  then  effectively  train  all  their  travelers  has  been  largely 
unsuccessful.  According  to  Navy  officials,  this  has  resulted  in  many 
travelers  and  users  attempting  to  use  DTS  with  no  or  insufficient  training. 
The  effect  has  frustrated  users  at  each  step  of  the  travel  process  and  has 
discouraged  use  of  DTS. 


•  Air  Force  officials  stated  that  new  DTS  system  releases  are  implemented 
with  known  problems,  but  the  sites  are  not  informed  of  the  problems. 
Workarounds  are  not  provided  until  after  the  sites  begin  encountering 
problems.  Air  Force  DTS  program  officials  stated  that  DTS  releases  did 
not  appear  to  be  well  tested  prior  to  implementation.  Air  Force  officials 
also  stated  that  there  was  insufficient  training  on  new  functionality.  PMO- 
DTS  and  DTS  contractor  program  officials  believed  that  conference  calls 
to  discuss  new  functionality  with  the  sites  were  acceptable  training,  but 
Air  Force  officials  did  not  agree.  The  Air  Force  finance  office  was 
expected  to  fully  comprehend  the  information  received  from  those 
conference  calls  and  provide  training  on  the  new  functionality  to 
users/approvers,  but  these  officials  stated  that  this  was  an  unrealistic 
expectation. 
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As  discussed  in  our  September  2005  testimony  and  January  2006  report, 
the  unnecessary  continued  use  of  the  legacy  travel  systems  results  in  the 
inefficient  use  of  funds  because  the  department  is  paying  to  operate  and 
maintain  duplicative  systems  that  perform  the  same  function — ^travel. 


Previously  Reported 
DTS  Requirements 
Management  and 
Testing  Deficiencies 
Have  Not  Been 
Resolved 


Our  September  2005  testimony  and  January  2006  report  noted  problems 
with  DTS’s  ability  to  properly  display  flight  information  and  traced  those 
problems  to  inadequate  requirements  management  and  testing.  DOD 
stated  that  it  had  addressed  those  deficiencies,  and  in  February  2006,  we 
again  tested  the  system  to  determine  whether  the  stated  weaknesses  had 
been  addressed.  We  found  that  similar  problems  continue  to  exist.  Once 
again,  these  problems  can  be  traced  to  ineffective  requirements 
management  and  testing  processes.  Properly  defined  requirements  are  a 
key  element  in  systems  that  meet  their  cost,  schedule,  and  performance 
goals  since  the  requirements  define  the  (1)  functionality  that  is  expected 
to  be  provided  by  the  system  and  (2)  quantitative  measures  by  which  to 
determine  through  testing  whether  that  functionality  is  operating  as 
expected. 


We  briefed  PMO-DTS  officials  on  the  results  of  our  tests  and  in  May  2006 
the  officials  agreed  that  our  continued  concerns  about  the  proper  display 
of  flight  information  were  valid.  PMO-DTS  officials  stated  that  the  DTS 
technology  refresh,  which  was  to  be  completed  in  September  2006,  should 
address  some  of  our  concerns.  While  these  actions  are  a  positive  step 
forward,  they  do  not  address  the  fundamental  problem  that  DTS’s 
requirements  are  still  ambiguous  and  conflicting — a  primary  cause  of  the 
previous  problems.  Until  a  viable  requirements  management  process  is 
developed  and  effectively  implemented,  the  department  (1)  cannot 
develop  an  effective  testing  process  and  (2)  will  not  have  reasonable 
assurance  the  project  risks  have  been  reduced  to  acceptable  levels. 


Providing  Complete  Flight 
Information  Has  Been  a 
Continuing  Problem 


In  our  earlier  testimony  and  report,^®  we  noted  that  DOD  did  not  have 
reasonable  assurance  that  the  flights  displayed  met  the  stated  DOD 
requirements.  Although  DOD  stated  in  each  case  that  our  concerns  had 
been  addressed,  subsequent  tests  found  that  the  problems  had  not  been 
corrected.  Requirements  represent  the  blueprint  that  system  developers 
and  program  managers  use  to  design,  develop,  and  acquire  a  system. 


^^GAO-05-998T  and  GAO-06-18. 
^®GAO-05-998T  and  GAO-06-18. 
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Requirements  should  be  consistent  with  one  another,  verifiable,  and 
directly  traceable^®  to  higher-level  business  or  functional  requirements.  It  is 
critical  that  requirements  be  carefully  defined  and  that  they  flow  directly 
from  the  organization’s  concept  of  operations  (how  the  organization’s  day- 
to-day  operations  are  or  will  be  carried  out  to  meet  mission  needs). 
Improperly  defined  or  incomplete  requirements  have  been  commonly 
identified  as  a  cause  of  system  failure  and  systems  that  do  not  meet  their 
cost,  schedule,  or  performance  goals. 

Requirements  represent  the  foundation  on  which  the  system  should  be 
developed  and  implemented.  As  we  have  noted  in  previous  reports,^” 
because  requirements  provide  the  foundation  for  system  testing, 
significant  defects  in  the  requirements  management  process  preclude  an 
entity  from  implementing  a  disciplined  testing  process.  That  is, 
requirements  must  be  complete,  clear,  and  well  documented  to  design  and 
implement  an  effective  testing  program.  Absent  this,  an  organization  is 
taking  a  significant  risk  that  its  testing  efforts  will  not  detect  significant 
defects  until  after  the  system  is  placed  into  production.  Our  February  2006 
analysis  of  selected  flight  information  disclosed  that  DOD  still  did  not  have 
reasonable  assurance  that  DTS  displayed  flights  in  accordance  with  its 
stated  requirements.  We  analyzed  15  U.S.  General  Services  Administration 
(GSA)  city  pairs,^‘  which  should  have  translated  into  246  GSA  city  pair 
flights  for  the  departure  times  selected.  However,  we  identified  87  flights 
that  did  not  appear  on  one  or  more  of  the  required  listings  based  on  the 
DTS  requirements.  For  instance,  our  analysis  identified  44  flights 


^^Traceability  allows  the  user  to  follow  the  life  of  the  requirement  both  forward  and 
backward  through  these  documents  and  from  origin  through  implementation.  Traceability 
is  also  critical  to  understanding  the  parentage,  interconnections,  and  dependencies  among 
the  individual  requirements.  This  information  in  turn  is  critical  to  understanding  the  impact 
when  a  requirement  is  changed  or  deleted. 

^”See,  for  example,  GAO-04-1008  wdArmy  Depot  Maintenance:  Ineffective  Oversight  of 
Depot  Maintenance  Operations  and  System  Implementation  Efforts,  GAO-05-441 
(Washington,  D.C.:  June  30,  2005). 

^*GSA  awards  contracts  to  airlines  to  provide  flight  services  between  pairs  of  cities.  This  is 
commonly  referred  to  as  the  GSA  city  pair  program.  Under  this  program  (1)  no  advanced 
ticket  purchases  are  required,  (2)  no  minimum  or  maximum  length  of  stay  is  required, 

(3)  tickets  are  fully  refundable  and  no  charges  are  assessed  for  cancellations  or  changes, 

(4)  seating  is  not  capacity  controlled  (i.e.,  as  long  as  there  is  a  coach-class  seat  on  the 
plane,  the  traveler  may  purchase  it),  (5)  no  blackout  dates  apply,  (6)  fare  savings  average 
70  percent  over  regular  walk-up  fares,  and  (7)  fares  are  priced  on  one-way  routes 
permitting  agencies  to  plan  for  multiple  destinations.  We  selected  the  first  16  city  pairs  that 
were  provided  by  DOD  to  GSA  in  support  of  a  GSA  study  on  accuracy  of  flight  displays  and 
fare  information  by  DTS  and  the  GSA  eTravel  providers. 
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appearing  on  other  DTS  listings  or  airline  sites  that  did  not  appear  on  the 
9:00  am  DTS  listing  even  though  those  flights  (1)  met  the  12-hour  flight 
window^^  and  (2)  were  considered  GSA  city  pair  flights — two  of  the  key 
DTS  requirements  the  system  was  expected  to  meet. 

After  briefing  PMO  officials  on  the  results  of  our  analysis  in  February  2006, 
the  PMO-DTS  employed  the  services  of  a  contractor  to  review  DTS  to 
determine  the  specific  cause  of  the  problems  and  recommend  solutions.  In 
a  March  2006  briefing,  the  PMO-DTS  acknowledged  the  existence  of  the 
problems,  and  identified  two  primary  causes.  First,  part  of  the  problem 
was  attributed  to  the  methodology  used  by  DTS  to  obtain  flights  from  the 
Global  Distribution  System  (GDS).  The  PMO-DTS  stated  that  DTS  was 
programmed  to  obtain  a  “limited”  amount  of  data  from  GDS  in  order  to 
reduce  the  costs  associated  with  accessing  GDS.  This  helps  to  explain  why 
flight  queries  we  reviewed  did  not  produce  the  expected  results.  To 
resolve  this  particular  problem,  the  PMO-DTS  proposed  increasing  the 
amount  of  data  obtained  from  GDS.  Second,  the  PMO-DTS  acknowledged 
that  the  system  testing  performed  by  the  contractor  responsible  for 
developing  and  operating  DTS  was  inadequate  and,  therefore,  there  was 
no  assurance  that  DTS  would  provide  the  data  in  conformance  with  the 
stated  requirements.  This  weakness  was  not  new,  but  rather  reconfirms 
the  concerns  discussed  in  our  September  2005  testimony  and  January  2006 
report^^  related  to  the  testing  of  DTS. 


DOD’s  Planned  Corrective 
Actions  Will  Not  Address 
Fundamental 

Requirements  Management 
Problems 


While  DOD’s  planned  actions,  including  a  recent  technology  upgrade, 
should  address  several  of  the  specific  weaknesses  we  identified  related  to 
flight  displays,  they  fall  short  of  addressing  the  fundamental  problems  that 
caused  those  weaknesses — inadequate  requirements  management.  DTS’s 
requirements  continue  to  be  ambiguous.  For  example,  DOD  has  retained  a 
requirement  to  display  25  flights  for  each  inquiry.  However,  it  has  not 
determined  (1)  whether  the  rationale  for  that  requirement  is  valid  and 
(2)  under  what  conditions  flights  that  are  not  part  of  the  GSA  city  pair 
program  should  be  displayed.  For  example,  we  found  that  several  DTS 


flight  window  is  the  amount  of  time  before  and  after  a  specified  time  and  is  used  for 
determining  the  flights  that  should  be  displayed.  DTS  uses  a  12-hour  flight  window  for 
domestic  flights  and  a  24-hour  flight  window  for  foreign  flights.  The  system  is  also 
expected  to  display  up  to  26  flights  for  the  flight  window. 

’’AAO-05-998T  and  GAO-06-18. 
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flights  displayed  to  the  user  “overlap other  flights.  Properly  validating 
the  requirements  would  allow  DOD  to  obtain  reasonable  assurance  that  its 
requirements  properly  define  the  functionahty  needed  and  the  business 
rules  necessary  to  properly  implement  that  functionality.  As  previously 
noted,  requirements  that  are  unambiguous  and  consistent  are  fundamental 
to  providing  reasonable  assurance  that  a  system  will  provide  the  desired 
functionality.  Until  DOD  improves  DTS  requirement  management 
practices,  it  will  not  have  this  assurance. 


Recommendations 
Improve  DTS 
Management  and 
Oversight 


Our  recent  report^^  included  four  recommendations  to  improve  the 
department’s  management  and  oversight  of  DTS.  We  recommended  that 
DOD  (1)  evaluate  the  cost  effectiveness  of  the  Navy  continuing  with  the 
CTO  management  fee  structure  versus  adopting  the  revised  CTO  fee 
structure,  once  the  new  contracts  have  been  awarded,  (2)  develop  a 
process  by  which  the  military  services  develop  and  use  quantitative  data 
from  DTS  and  their  individual  legacy  systems  to  clearly  identify  the  total 
universe  of  DTS-eligible  transactions  on  a  monthly  basis,  (3)  require  the 
PMO-DTS  to  provide  periodic  reports  on  the  utilization  of  DTS,  once 
accurate  data  are  available,  and  (4)  resolve  inconsistencies  in  DTS 
requirements  by  properly  defining  the  functionality  needed  and  business 
rules  necessary  to  properly  implement  the  needed  functionality.  DOD 
concurred  with  three  and  partially  concurred  with  one  of  the 
recommendations.  In  regard  to  the  recommendations  with  which  the 
department  concurred,  it  briefly  outlined  the  actions  it  planned  to  take  in 
addressing  two  of  the  three  recommendations.  For  example,  the 
department  noted  the  difficulties  in  obtaining  accurate  utilization  data 
from  the  existing  legacy  systems,  but  stated  that  the  Office  of  the  Under 
Secretary  of  Defense  (Personnel  and  Readiness)  and  BTA  will  evaluate 
methods  for  reporting  actual  DTS  utilization. 


Additionally,  DOD  noted  that  the  Defense  Travel  Management  Office 
developed  and  implemented  a  requirements  change  management  process 


^‘*For  example,  DTS  displayed  a  GSA  city  pair  flight  between  Washington,  D.C.,  and  Atlanta, 
Georgia,  that  departed  at  10:05  a.m.  and  arrived  at  1:50  p.m.  This  flight  “overlapped”  two 
other  GSA  city  pair  direct  flights  that  were  available  and  required  less  travel  time.  One 
flight  left  at  10:05  a.m.  and  arrived  at  12:02  p.m.  while  another  left  at  11:05  a.m.  and  arrived 
at  12:56  p.m.  Furthermore,  DTS  displayed  a  non-GSA  city  pair  flight  that  left  at  9:20  a.m. 
and  arrived  at  1:05  p.m.  This  flight  did  not  meet  any  of  the  acceptable  criteria  for  not  using 
a  GSA  city  pair  flight. 

^^GAO-06-980. 
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on  May  1,  2006.  In  commenting  on  the  report,  the  department  stated  that 
this  process  is  intended  to  define  requirements  and  track  the  entire  life 
cycle  of  the  requirements  development  process.  While  we  fully  support  the 
department’s  efforts  to  improve  its  management  oversight  of  DTS’s 
requirements,  we  continue  to  believe  that  the  department  needs  to  have  in 
place  a  process  that  provides  DOD  reasonable  assurance  that 
(1)  requirements  are  properly  documented  and  (2)  requirements  are 
adequately  tested  as  recommended  in  our  January  2006  report.^®  This 
process  should  apply  to  all  existing  requirements  as  well  as  any  new 
requirements.  As  discussed  in  this  report,  we  reviewed  in  May  2006  some 
of  the  requirements  that  were  to  have  followed  the  new  requirements 
management  process  and  formd  problems  similar  to  those  noted  in  our 
January  2006  report.  Although  we  did  not  specifically  review  the  new 
process,  if  it  does  not  include  an  evaluation  of  existing  requirements,  the 
department  may  continue  to  experience  problems  similar  to  those  we 
previously  identified. 

DOD  partially  concurred  with  our  recommendation  to  evaluate  the  cost 
effectiveness  of  the  Navy  continuing  with  the  CTO  management  fee 
structure.  However,  DOD’s  response  indicated  that  the  Defense  Travel 
Management  Office  is  currently  procuring  commercial  travel  services  for 
DOD  worldwide  in  a  manner  that  will  ensure  evaluation  of  cost 
effectiveness  for  all  services.  If  DOD  proceeds  with  the  actions  outlined  in 
its  comments,  it  will  meet  the  intent  of  our  recommendation. 

Effective  implementation  of  these  recommendations  as  well  as  those 
included  in  our  January  2006  report^^  will  go  a  long  way  towards  improving 
DTS  functionahty  and  increasing  utilization.  Furthermore,  the  shift  of  DTS 
to  the  BTA,  which  makes  DTS  an  enterprisewide  endeavor,  should  help  in 
making  DTS  the  standard  integrated,  end-to-end  travel  system  for  business 
travel.  Management  oversight  is  essential  for  this  to  become  a  reality.  As  I 
stated  previously,  in  written  comments  on  a  draft  of  our  report,  the  Under 
Secretary  of  Defense  (Personnel  and  Readiness),  strongly  objected  to  our 
finding  that  the  estimated  personnel  savings  included  in  the  economic 
analysis  are  unrealistic.  Because  none  of  the  military  services  could 
validate  an  actual  reduction  in  the  number  of  personnel  as  a  result  of  DTS 
implementation,  and  DOD’s  comments  did  not  include  any  additional 
support  or  dociimentation  for  its  position,  we  continue  to  believe  that  the 


^®GAO-06-18. 

^^GAO-06-18. 


Page  21 


GAO-07-208T 


estimated  annual  personnel  savings  of  $54.1  million  are  unrealistic. 
Although  the  department’s  criteria  do  not  require  that  a  new  economic 
analysis  be  prepared,  the  fiscal  year  2005  defense  authorization  acf^ 
requires  the  periodic  review,  but  not  less  than  annually,  of  every  defense 
business  system  investment.  If  effectively  implemented,  this  annual  review 
process  provides  an  excellent  opportunity  for  DOD  management  to  assess 
whether  DTS  is  meeting  its  planned  cost,  schedule,  and  fimctionality  goals. 
Going  forward,  such  a  review  could  serve  as  a  useful  management  tool  in 
making  funding  and  other  management  decisions  related  to  DTS. 

In  conclusion,  overhauling  the  department’s  antiquated  travel  management 
practices  and  systems  has  been  a  daunting  challenge  for  DOD.  While  it 
was  widely  recognized  that  this  was  a  task  that  needed  to  be  accomplished 
and  savings  could  result,  the  underlying  assumptions  in  support  of  those 
savings  are  not  based  on  reliable  data  and  therefore  it  is  questionable 
whether  the  anticipated  savings  will  materialize.  Even  though  the  overall 
savings  are  questionable,  the  successful  implementation  of  DTS  is  critical 
to  reducing  the  number  of  stovepiped,  duplicative  travel  systems 
throughout  the  department.  We  have  reported  on  numerous  occasions  that 
reducing  the  number  of  business  systems  within  DOD  can  translate  into 
savings  that  can  be  used  for  other  mission  needs.  As  noted  above, 
management  oversight  will  be  an  important  factor  in  DTS  achieving  its 
intended  goals.  Equally  important,  however,  will  be  the  department’s 
ability  to  resolve  the  long-standing  difficulties  that  DTS  has  encountered 
with  its  requirements  management  and  system  testing.  Until  these  issues 
are  resolved,  more  complete  utilization  of  DTS  will  be  problematic. 


Mr.  Chairman,  this  concludes  my  prepared  statement.  We  would  be  happy 
to  answer  any  questions  that  you  or  other  members  of  the  Subcommittee 
may  have  at  this  time. 


Contacts  and 
Acknowledgments 


For  further  irrformation  about  this  testimony,  please  contact  McCoy 
Williams  at  (202)  512-9095  or  williamsml@gao.gov,  or  Keith  A.  Rhodes  at 
(202)  512-6412  or  rhodesk@gao.gov. 


^®Ronald  W.  Reagan  National  Defense  Authorization  Act  for  Fiscal  Year  2006,  Pub.  L.  No. 
108-375,  §  332,  118  Stat.  1811,  1851-56  (Oct.  28,  2004)  (codified,  in  part,  at  10  U.S.C.  §§  186, 
2222). 


Page  22 


GAO-07-208T 


Contact  points  for  our  Offices  of  Congressional  Relations  and  Public 
Affairs  may  be  found  on  the  last  page  of  this  report.  In  addition  to  the 
above  contacts,  the  following  individuals  made  key  contributions  to  this 
testimony:  Darby  Smith,  Assistant  Director;  J.  Christopher  Martin,  Senior- 
Level  Technologist;  F.  Abe  Dymond,  Assistant  General  Counsel;  Beatrice 
Alff;  Harold  Brumm,  Jr.;  Francine  DelVecchio;  and  Tarunkant  Mithani. 


(195104) 


Page  23 


GAO-07-208T 
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